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APPARAT US RES PONSIVE TO MOVEM ENX-Q^A-^Aq^N^ 
PURING TREATMENT/DIAGNOSIS^ 

BACKGROUND OF THE INVENTION 

Field of the Invention 

This invention relates to medical use of radiation for treatment and 
diagnosis, and more particularly to detection and response to patient movement during 
radiological treatment and diagnosis. 
Background Information 

Conventional radiotherapy treatment relies on simple patient setup 
techniques. These techniques use stationary and a limited number of radiation fields, 
which are often much wider than the tumor or volume, thus effectively compensating 
for the possibility of a tumor geometric miss. Consequently, a substantial amount of 
healthy tissue is irradiated and becomes a radio-biological dose limiting factor in tumor 
control. 

Modem conformal dynamic radiotherapy attempts to overcome the above 
radio-biological limitation by tight-margin conformation of radiation dose distribution 
tailored to the three-dimensional tumor volume by the use of computer-control multi- 
beam conformal dynamic radio therapy (CCRT). Consequently, the accuracy in patient 
position, knowledge of the movement of a patient including substantial motion of 
internal organs such as with breathing is of primary importance. In addition to patient 
movement which would cause the tight beam to miss the tumor, it is important to be 
able to detect patient movement which could cause a collision between the patient and 
the linear accelerator, which is repeatedly repositioned to establish the multiple 
treatment beams. 
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There is a need therefore for apparatus for detecting patient movement 
on radiological treatments/diagnostic equipment. 

There is a particular need for such apparatus which can detect sub- 
millimeter patient movement in real time. 

There is also a need for such apparatus which can detect patient 
movement initiated from various treatment positions. 

There is also a need for such apparatus which can detect patient 
movement under varying lighting conditions. 

There is a further need for such apparatus which can discriminate 
movement associated with patient breathing from other movement and accommodate 
therefor. 

SUMMARY OF THE INVENTION 

These needs and others are satisfied by the invention which is directed 
to apparatus responsive to movement of a patient which identifies and tracks movement 
of at least one passive fiducial on the patient. The apparatus applies multiple levels of 
filtering which can include: correlation, preferably normalized correlation, sparse 
sampling, bracketing and interpolation, and minima suppression to rapidly identify the 
location of the at least one fiducial. The multiple levels of filtering are applied at 
multiple levels of resolution of the digital image signals. 

Interest operators can be used in combination with templates to locate 
the positions of the passive fiducials. The templates can be selected interactively by 
a user from a display generated by the digital image signals. Alternatively, the 
template used for tracking is selected from images generated using an initial template. 
Rather than using the image which best matches the initial template, the template with 
a median match is selected. 

As another aspect of the invention, the means generating an output 
includes means indicating movement of the at least one passive fiducial relative to at 
least one selected level of displacement. Preferably, the output means generates a 
warning that movement exceeds a first displacement and includes means providing a 
signal for terminating radiation treatment when the movement exceeds a second greater 
displacement. Preferably, the means providing an indication of movement includes a 
display generating an image of the patient and the fiducials, together with an indication 
of movement relative to the first and second displacements. 
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As yet another aspect of the invention, the means determining movement 
of the passive fiducials includes means detecting movement associated with patient 
breathing and random movement. The movement associated with patient breathing can 
be used to generate a gating signal synchronized to patient breathing. This gating 
5 signal can then be used to actuate the beam generator only during selected parts of the 
breathing cycle. 

BRIEF DESCRIPTION OF THE DRAWINGS 
A full understanding of the invention can be gained from the following 
description of the preferred embodiments when read in conjunction with the 
10 accompanying drawings in which: 

Figure 1 is an isometric view of apparatus in accordance with the 
invention for implementing conformal dynamic radiotherapy. 

Figure 2 is a plan view of a patient reclining on a couch which forms 
part of the apparatus of Figure 1 and illustrating the placement of fiducials in 
15 accordance with the invention. 

Figure 3 is a perspective view of a preferred fiducial used in 
implementation of the invention. 

Figure 4 is a functional diagram illustrating implementation of the 

invention. 

20 Figure 5 is an illustration of a display which is generated by the 

apparatus of Figure 1 in implementation of the invention. 

Figures 6-16 are flow charts of software used in implementation of the 

invention. 

Figure 17 is an illustration of an interest operator which can be used in 
25 implementation of the invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 
Figure 1 illustrates a radiotherapy treatment system 1 in which the 
invention is implemented. This system 1 includes a machine 3 having a gantry 5 
pivotally mounted on a machine base 7 for rotation about a horizontal axis 9. The 
30 gantry 5 has a first arm 11 carrying a collimator 13 which directs a beam of high 
energy radiation 15, such as a beam of high energy photons, along a path which is 
perpendicular to and passes through an extension of the axis of rotation 9. This 
intersection is referred to as the isocenter 17. In some machines, a portal imager 19 
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is mounted on a second arm 21 on the opposite end of the gantry in alignment with the 
radiation beam 15. The portal imager 19 records radiation which is not absorbed by 
the patient. 

The isocenter 17 serves as the origin of a coordinate system for room 
space. As can be seen, the X axis coincides with the axis of rotation 9 of the gantry. 
Thus, as the gantry 5 rotates it defines a plane of treatment containing the Y and Z 
axes. 

The machine 3 further includes a patient positioning assembly 23, which 
includes a couch 25 mounted on a support 27 for vertical, lateral and longitudinal 
movement relative to the support. The support 27 is mounted on a turntable 29, which 
has its axis 31 vertically aligned under the isocenter 17 and concentric with the Z axis. 
With this arrangement, the patient positioning assembly 23 has four degrees of 
freedom: translation in the X, Y and Z axes of room space and rotation about the Z 
axis. Thus, the patient is not rotated about the longitudinal axis of the couch or tilted 
about a horizontal axis extending transversely through the couch. However, with the 
addition of rotation of the gantry in the Y-Z treatment plane, the radiation beam 15 can 
be directed through a patient reclining on the couch 25 in any desired direction. A 
computer 33 controls movement of the patient positioning assembly 23 and the gantry 
5 for establishing the progression of high energy treatment beams used in practicing 
conformal radiation therapy. 

As previously discussed, in conformal radiation therapy the beam 15 is 
tightly conformed by the collimator 13 to the specific tumor to be treated. Thus, 
movement of the patient on the couch 25 of the patient position assembly 23 can cause 
misalignment of the radiation beam 15 with the tumor. This not only degrades 
treatment of the tumor but also exposes surrounding healthy tissue to unwanted levels 
of radiation. In addition, normal breathing by the patient can cause movement of 
internal organs by an amount which would result in misalignment of the beam. For 
instance, a tumor on the lower portion of the lung can move several centimeters during 
normal breathing. Slight movement of the patient can be tolerated; however, treatment 
should be terminated if acceptable tolerances of movement are exceeded. Furthermore, 
excessive movement by the patient can also cause a collision between the patient and 
the gantry as the patient positioning assembly 23 and gantry are positioned for 
successive treatment beams. 
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The invention employs a vision system 34 to measure and respond to 
patient movement. The vision system 34 includes at least one video camera 35. 
Preferably, multiple cameras are used. In the exemplary embodiment of the invention 
a first camera 35 1 is mounted on the first arm 1 1 of the gantry 5 adjacent the collimator 
5 13 and is aimed to capture an image of a patient 37 positioned on the couch 25, as 
shown in Figure 2. As the camera 35, will be below the couch 25 for some positions 
of the gantry 5, a second camera 352 is fixed to the ceiling over the patient positioning 
assembly 23. The field of view of this camera 352 will be blocked when the gantry 5 
is at the top of its arc. Thus, the patient is visible to at least one camera 35 at all 

10 times. Additional cameras 35 could be provided, such as cameras laterally displaced 
from the patient positioning assembly 23 to provide more sensitivity to movement along 
the axis of, for instance, the camera 352- However, as will be discussed below, a 
single camera can detect three-dimensional movement, including movement toward and 
away from the camera which is detected as a change in the size of the image. 

15 In the exemplary embodiment of the invention, natural or artificial 

fiducials are used to detect patient movement. Natural fiducials could be scars or other 
prominent features of the patient. The preferred fiducial 39 shown in Figure 3 is a 
sphere 41 covered with a material having a lambertian surface. Such a surface is 
highly reflective under low light conditions, yet provides a uniform scattered reflection 

20 with no highlights. The sphere 41 is attached to the center of a non-reflective base 43 
which is secured to the patient's skin, such as by an adhesive. 

In principle, only one fiducial 39 is required. As a practical matter, it 
is advantageous to provide multiple fiducials placed on the patient so as to detect any 
movement of the critical locations. Thus, as shown in Figure 2, by way of example, 

25 four fiducials 39 are placed on the patient's chest. Natural skin markings could be 
used in addition to the artificial fiducials shown in Figure 3. If more than one 
camera 35 is used, each tracks as many of the fiducials 39 as it can see. 

Figure 4 is a functional diagram of the invention. The camera(s) 35 
capture an image of the fiducials 39 on the patient 37 reclining on the patient 

30 positioning assembly 23. The image captured by the camera 35 is digitized by digitizer 
45 to generate digital image signals. These digital image signals are 0 to 255 gray 
scale signals for each camempi^l. The digital image ^^^^ processed by a 

/(!) processor which includes a .pesrtiofi motion detector 47. ^^i^^n motion detector 47 
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is implemented in the computer 49 shown in Figure 1 . The computer 49 includes a 
monitor 51 which generates a display 53, an example of which is shown in Figure 5. 
The man machine interface 55 for the computer 49 includes a keyboard 57 and a 
pointing device 59, such as a mouse or trackball. 
5 As will be discussed fully, the patient motion detector 47 detects and 

identifies the fiducials 39 and then tracks their movement. Movement within a certain 
narrow tolerance is acceptable, while larger movements are unacceptable. Visible 
and/or audio warnings of these two classifications of movement can be generated. A 
gating signal generator 61 responds to unacceptable movement to disable the beam 
10 generator 63. This unacceptable movement which would terminate the radiation beam 
can be movement which displaces the target tumor so that it is missed by the radiation 
beam, or could be movement which would cause a collision between the patient and 
the gantry 5 during movement of the machine from one treatment beam to the next. 
In the former case, the gating signal generator 61 could re-enable the beam generator, 
15 if the patient returns to the proper position. For instance, a large sigh could 
temporarily displace the target area by an unacceptable amount. In accordance with 
another aspect of the invention, the patient motion detector 47 can track patient 
breathing and extract such quasi-periodic motion from random patient motion. Gating 
of the beam generator can then be synchronized with patient breathing. For instance, 
20 a tumor on the lung could move up to 4 to 5 centimeters during patient breathing. 

This is an unacceptable amount of movement. However, by synchronizing generation 
of the radiation beam with breathing, the tumor can be repetitively irradiated at a fixed 
position during the breathing cycle. 

As shown in Figure 5, the display 53 presents an image of the patient 
25 37 with the fiducials 39 appearing prominently. An indicator 65, such as the square 
shown, surrounds each fiducial and is color coded to indicate the state of motion of the 
fiducial. The fiducial with the largest displacement such as 39a is singled out by a 
distinctive marker, such as a red square 65a, while the remaining markers are green 
squares in the exemplary system. The display also includes a traffic light 67 having 
30 a green section 67g, a yellow section 67y and a red section 67r. When motion of the 
fiducials is within preferred tolerances, the green section 67g of the traffic light is on. 
For motion which is outside the normal range, but which is still acceptable, the yellow 
section 67y is on. The traffic light turns red when the motion of any of the fiducials 
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is approaching the unacceptable. A scale 69 along the side of the display 53 indicates 
in bar graph form the percentage of maximum allowable displacement of the fiducial 
of maximum displacement. Thus, for instance, if the red light 67r is illuminated and 
the bar graph 71 indicates 80%, the fiducial with maximum displacement has moved 
5 by a distance which is four fifths of the way through the acceptable displacement. The 
green, yellow and red regions need not be equal as shown in the example. 

Detection of motion of a patient using passive fiducials requires an 
implementation which is robust enough to accommodate for the variations in the 
shapes, appearance and lighting conditions to which the fiducials are subjected and, at 

10 the same time, is fast enough to provide real time tracking of patient movement. The 
invention satisfies these requirements by utilization of successive levels of filtering and 
templates which are modified to accommodate for actual conditions. The result is a 
system which can track patient movement at 20 Hz or better. 

Flow charts of suitable software 100 for implementing the invention are 

15 illustrated in Figures 6-16. Figure 6 illustrates the main routine of the software 100 
and includes detecting fiducials on the patient's body is in the current camera image 
at 110. As will be described, this is accomplished utilizing templates. The templates 
are then fine tuned at 120 for the specific patient and environmental conditions. As 
long as the user desires monitoring as determined at 130, a loop is entered in which 

20 each individual fiducial is tracked as indicated at 140. It is possible that a fiducial can 
be lost by the tracking system. This could occur, for instance, if the patient moves so 
that a fiducial is blocked from the camera's view, or the patient moves a hand through 
the line of sight of the camera. Also, a fiducial may be temporarily lost by rapid 
movement or adverse lighting conditions. If a fiducial is lost, as determined at 150, 

25 a number of attempts can be made to reacquire it. If the fiducial is not reacquired 
within a reasonable time, however, it is removed from tracking as indicated by 160 and 
170. If the selected number of attempts to reacquire, such as for example, five, have 
not been reached, an attempt is made to reacquire the fiducial at 180. If the fiducial 
is reacquired at 190, then a routine is run at 200 to generate any alarm if needed, and 

30 gating signals for the accelerator or beam generator 63 as indicated at 200. As long 
as any fiducials remain to be tracked as indicated at 210, the tracking loop is 
repetitively run. 
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Figure 7 illustrates the general routine 110 for detecting the fiducials 39 
in the image represented by the digital image signals. As mentioned, templates are 
used to identify the locations of the fiducials. The templates indicate what the pattern 
of digital signals representing the fiducial should look like. The size of the templates 
used must be considered. Larger templates improve the accuracy but take longer to 
process. In the exemplary system, templates 40 pixels square have been utilized. 
There are several ways in which the templates can be generated. As indicated at 111 
in Figure 7, idealized image templates can be utilized. In addition to such idealized 
templates or in place thereof, pre-stored image templates for the patient can be used 
as indicated at 112. Such pre-stored templates are used, for instance, for natural 
fiducials such as scars. One template is used for each family of fiducials. For 
instance, if all of the fiducials are the preferred fiducials such as shown in Figure 3, 
only one template is required because all of the fiducials in the family will generate a 
similar image. 

In addition, templates can be selected interactively by the user at 113, 
This is accomplished by using the mouse or trackball 59 to click on the center of a 
representation of the fiducial on the display 53. 

Where the idealized or pre-stored templates are utilized, a multi- 
resolution pyramid is used to locate the fiducials in the image using the templates. 
Thus, as indicated at 114, a search is made of the current image in low resolution for 
candidate matches of all template families. In the exemplary embodiment of the 
invention, one-third resolution is used at this point. Matches are made using a 
normalized correlation between the template and the image. The matches found in low 
resolution are then verified and localized in high resolution at 115. The K best 
matches are then selected as the most reliable fiducials at 116 where K equals the 
number of fiducials to be tracked. The user is then given the opportunity at 1 17 to edit 
the detected location of fiducials found either through use of the idealized or pre-stored 
templates or templates generated interactively. 

The details of the low resolution detection routine performed in block 
114 of Figure 7 is shown in Figure 8. As shown at 114.1, the image can be raster 
scanned selecting points using sparse sampling. In raster scanning pixels are 
considered successively along each line, line-by-line in increments of one, while in 
sparse sampling the increment is greater than one. Alternatively, the image can be 
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raster scanned as indicated at 114.2, selecting candidate points using interest operators 
followed by thresholding. Interest operators are simple patterns which emphasize gray 
scale characteristics of a particular fiducial. An example is shown in Figure 17, where 
the fiducial is a light circle 73 on a dark background 75. The interest operator 77 
5 could be, for instance, the one pixel value 79 in the center having a gray scale value 
matching that of the light circle 73, and the four pixels 81 at the cardinal points having 
gray scale values similar to that of the background 75. Such interest operators permit 
rapid searching of the image and should be selected as to assure identifying all of the 
fiducials in the family. They will most likely also generate additional candidate points. 
10 Returning to Figure 8, the interest operator generated value in the exemplary system 
is the relative albedo. The relative albedo of each point in the low resolution scan is 
compared to a threshold value to select candidate points. 

For each candidate point, a template matching is performed at 114.3, 
using a normalized correlation. Unwanted point matches are then filtered out at 1 14.4 
15 using thresholding on the normalized correlation value. In the exemplary embodiment, 
a normalized correlation of 0.75 was used as the threshold. Bracketing and 
interpolation are then used at 114.5 to localize the remaining point/ matches. In 
implementing bracketing, a rectangular image window is selected within which the 
desired point match will definitely lie. Then by interpolating between the correlation 
20 values of points on the border of the selected window along with its center, a new 
estimate of the location of the point match is calculated. This process is repeated with 
successively smaller windows centered on the new estimate of the location of the point 
match until a singular point is reached. In the exemplary system, the interpolation is 
performed using a two-dimensional Gaussian distribution. 
25 Figure 9 illustrates the techniques for verifying the candidate matches 

in high-resolution indicated at 115 in Figure 7. Bracketing is performed on the 
selected matches in high resolution as indicated at 1 15. 1 . These points are then filtered 
at 115.2 within the same image neighborhood using minima suppression. In 
implementing minima suppression, for each point which has been a match, an area the 
30 size of the template is centered on the point. A point is selected as a further candidate 
match only if it is the best correlation with the template within the template window. 

An important aspect of the invention is the fine tuning of the tracking 
templates called for at 120 in Figure 6. Figure 10 illustrates the details of fine tuning 
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the templates. As indicated at 121, the median point/match from fiducials detected 
using the same initial template is selected. For example, if there are three point 
matches for a fiducial family, the match having the middle value of correlation is 
selected. Notice that the match with the best correlation is not selected as it is likely 
5 to eliminate some valid matches. This technique adapts the selection of the template 
to be used for tracking to the actual conditions existing at the time of the selection. 
The relevant image portion is then acquired as the new template at 122, and the 
position, the interest operator value and the normalized correlation for all relevant 
point/matches using this newly acquired template is then recorded at 123. The steps 

10 121-123 are accomplished for each template family. Then, the current spacial pattern 
of all the fiducials determined by the point/matches, is recorded at 124. 

The program then enters the tracking loop at block 130 in Figure 6. The 
routine for continuous tracking, which is called at 140 in Figure 6 is illustrated in 
Figure 11. The new position of the fiducial is estimated at 131 by projecting a velocity 

15 vector calculated from prior positions of the fiducial. Localization of fiducial position 
is then implemented in low resolution using bracketing and interpolation as indicated 
at 132. This is followed by high resolution localization of the fiducial position at 133, 
also using bracketing and interpolation. 

The low resolution localization of block 131 is implemented by the 

20 routine illustrated in Figure 12. As indicated at 132.1 points are selected by raster 
scanning the image window using sparse sampling. If interest operators are used, the 
interest operators with the value closest to that of the fiducial in the previous tracking 
step is selected at 132.2. In either case, a best match is selected using normalized 
correlation template matching at 132.3. This is followed by bracketing on the position 

25 of the best match at 132.4. 

Figure 13 illustrates the high resolution localization of fiducials called 
for in block 133 of Figure 11. As indicated, bracketing is performed on a candidate 
with best match in high resolution as indicated at 133.1. If a match is found, the 
normalized correlation, interest operator value and position of the best match are 

30 calculated at 133.2. If desired, the sub-pixel accuracy of the position can be calculated 
at 133.3. The same interpolation technique as in bracketing and interpolation, as 
described above, is used. Alternatively, bilinear interpolation between the surrounding 
pixel correlation values could be used. Finally, if needed, charge coupled device 
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(CCD) jitter is filtered out of the position at 133.4. In the exemplary system, a low 
pass filter is used. 

The lost fiducial routine 150 in Figure 6 is shown in Figure 14. If the 
tracking routine finds no fiducial within the specified image window at 151, then 
5 clearly the fiducial has been lost. Even if a fiducial has been found, confirmation must 
be made that it is in fact the new position of the fiducial. Hence, a number of 
constancy tests are applied in 152. For instance, the normalized correlation value and 
the interest operator value must not change by more than a selected amount, such as, 
for example, 15%, from the most current values. Also, image limits are applied. For 

10 instance, the fiducial should not have changed position by more than a predetermined 
amount or, if the edge of the image is reached, the position indicated is not accepted 
as the fiducial may be out of the field of view, although a continued indication that it 
is at the edge may be presented. 

The routine 180 in Figure 6 for reacquiring the lost fiducial is shown in 

15 Figure 15. First, the new position of the fiducial is estimated at 181 using a larger 
search window than was used at 141 in Figure 11. The image window is then raster 
scanned in high resolution using sparse sampling to select the best match, if any, at 

182. Bracketing is then performed around the position of the best match, if any, at 

183. The normalized correlation interest operator albedo and the position of the 
20 fiducial best matched is then determined at 184. This is followed by calculation of 

sub-pixel accuracy, if needed, at 185. Finally, the number of successive attempts to 
reacquire the fiducial is updated at 186. 

Figure 16 illustrates the routine 200 in Figure 6 for generating the 
alarms and gating the accelerator or beam generator. The direction and distance 

25 traveled by each currently actively tracked fiducial since the detection step is estimated 
at 201. The spacial pattern of the actively tracked fiducials is compared with the initial 
pattern and previous patterns at 202. Any quasi-periodic motion associated with the 
individual fiducials and/or the spacial pattern is predicted at 203 such as by using past 
data analysis. This would include movement associated with breathing or tremor of 

30 the patient. The alarm warnings, alarm states and accelerator gating signals are then 
computed at 204 for display or for feedback to the equipment, such as the accelerator. 

While specific embodiments of the invention have been described in 
detail, it will be appreciated by those skilled in the art that various modifications and 
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alternatives to those details could be developed in light of the overall teachings of the 
disclosure. Accordingly, the particular arrangements disclosed are meant to be 
illustrative only and not limiting as to the scope of invention which is to be given the 
full breadth of the claims appended and any and all equivalents thereof. 



